Assessment of oral proficiency in EAP programs : a case for pair interaction

Reliability and validity in the testing of spoken language are essential in order to assess learners' English language proficiency as evidence of their readiness to begin courses in tertiary institutions. Research has indicated that the task chosen to elicit language samples can have a marked effect on both the nature of the interaction, including the power differential, and assessment, raising the issue of ethics. This exploratory studey, with a group of 32 students from the Peoples's Republic of China preparing for tertiary study in Singapore, compares test-takers' reactions to the use of an oral proficiency interview and a pair interaction.

Transposing Musical Skill: Sonification of movement as concurrent augmented feedback enhances learning in a bimanual task

Concurrent feedback provided during acquisition can enhance performance of novel tasks. The ‘guidance hypothesis’ predicts that feedback provision leads to dependence and poor performance in its absence. However, appropriately-structured feedback information provided through sound (‘sonification’) may not be subject to this effect. We test this directly using a rhythmic bimanual shape-tracing task in which participants learned to move at a 4:3 timing ratio. Sonification of movement and demonstration was compared to two other learning conditions: (1) sonification of task demonstration alone and (2) completely silent practice (control). Sonification of movement emerged as the most effective form of practice, reaching significantly lower error scores than control. Sonification of solely the demonstration, which was expected to benefit participants by perceptually unifying task requirements, did not lead to better performance than control. Good performance was maintained by participants in the sonification condition in an immediate retention test without feedback, indicating that the use of this feedback can overcome the guidance effect. On a 24-hour retention test, performance had declined and was equal between groups. We argue that this and similar findings in the feedback literature are best explained by an ecological approach to motor skill learning which places available perceptual information at the highest level of importance.

Implementation of an audit with feedback knowledge translation intervention to promote medication error reporting in health care: a protocol.

BACKGROUND: Health professionals strive to deliver high-quality care in an inherently complex and error-prone environment. Underreporting of medical errors challenges attempts to understand causative factors and impedes efforts to implement preventive strategies. Audit with feedback is a knowledge translation strategy that has potential to modify health professionals' medical error reporting behaviour. However, evidence regarding which aspects of this complex, multi-dimensional intervention work best is lacking. The aims of the Safe Medication Audit Reporting Translation (SMART) study are to: 1. Implement and refine a reporting mechanism to feed audit data on medication errors back to nurses 2. Test the feedback reporting mechanism to determine its utility and effect 3. Identify characteristics of organisational context associated with error reporting in response to feedback METHODS/DESIGN: A quasi-experimental design, incorporating two pairs of matched wards at an acute care hospital, is used. Randomisation occurs at the ward level; one ward from each pair is randomised to receive the intervention. A key stakeholder reference group informs the design and delivery of the feedback intervention. Nurses on the intervention wards receive the feedback intervention (feedback of analysed audit data) on a quarterly basis for 12 months. Data for the feedback intervention come from medication documentation point-prevalence audits and weekly reports on routinely collected medication error data. Weekly reports on these data are obtained for the control wards. A controlled interrupted time series analysis is used to evaluate the effect of the feedback intervention. Self-report data are also collected from nurses on all four wards at baseline and at completion of the intervention to elicit their perceptions of the work context. Additionally, following each feedback cycle, nurses on the intervention wards are invited to complete a survey to evaluate the feedback and to establish their intentions to change their reporting behaviour. To assess sustainability of the intervention, at 6 months following completion of the intervention a point-prevalence chart audit is undertaken and a report of routinely collected medication errors for the previous 6 months is obtained. This intervention will have wider application for delivery of feedback to promote behaviour change for other areas of preventable error and adverse events.

Proving My Competency One Test at a Time: Internationally Educated Nurses and the Canadian English Language Benchmark Assessment for Nurses

Abstract Professional language assessment is a new concept that has great potential to benefit Internationally Educated Professionals and the communities they serve. This thesis reports on a qualitative study that examined the responses of 16 Canadian English Language Benchmark Assessment for Nurses (CELBAN) test-takers on the topic of their perceptions of the CELBAN test-taking experience in Ontario in the winter of 2015. An Ontario organization involved in registering participants distributed an e-mail through their listserv. Thematic analyses of focus group and interview transcripts identified 7 themes from the data. These themes were used to inform conclusions to the following questions: (1) How do IENs characterize their assessment experience? (2) How do IENs describe the testing constructs measured by the CELBAN? (3) What, if any, potential sources of construct irrelevant variance (CIV) do the test-takers describe based on their assessment experience? (4) Do IENs feel that the CELBAN tasks provide a good reflection of the types of communicative tasks required of a nurse? Overall, participants reported positive experiences with the CELBAN as an assessment of their language skills, and noted some instances in which they felt some factors external to the assessment impacted their demonstration of their knowledge and skill. Lastly, some test-takers noted the challenge of completing the CELBAN where the types of communicative nursing tasks included in the assessment differed from nursing tasks typical of an IENs country or origin. The findings are discussed in relation to literature on high-stakes large-scale assessment and IEPs, and a set of recommendations are offered to future CELBAN administration. These recommendations include (1) the provision of a webpage listing all licensure requirements (2) monitoring of CELBAN location and dates in relation to the wider certification timeline for applicants (3) The provision of additional CELBAN preparatory materials (4) Minor changes to the CELBAN administrative protocols. Given that the CELBAN is a relatively new assessment format and its widespread use for high-stakes decisions (a component of nursing certification and licensure), research validating IEN-test-taker responses to construct representation and construct irrelevant variance is critical to our understanding of the role of competency testing for IENs.

Item and task difficulty in a B2 reading examination: perceptions of test-takers and CEFR alignment experts compared with psychometric measurements

The article presents a study of a CEFR B2-level reading subtest that is part of the Slovenian national secondary school leaving examination in English as a foreign language, and compares the test-taker actual performance (objective difficulty) with the test-taker and expert perceptions of item difficulty (subjective difficulty). The study also analyses the test-takers’ comments on item difficulty obtained from a while-reading questionnaire. The results are discussed in the framework of the existing research in the fields of (the assessment of) reading comprehension, and are addressed with regard to their implications for item-writing, FL teaching and curriculum development.

The State of our Toolbox: A Meta-analysis of Reliability Measurement Precision

My study investigated internal consistency estimates of psychometric surveys as an operationalization of the state of measurement precision of constructs in industrial and organizational (I/O) psychology. Analyses were conducted of samples used in research articles published in the Journal of Applied Psychology between 1975 and 2010 in five year intervals (K = 934) from 480 articles yielding 1427 coefficients. Articles and their respective samples were coded for test-taker characteristics (e.g., age, gender, and ethnicity), research settings (e.g., lab and field studies), and actual tests (e.g., number of items and scale anchor points). A reliability and inter-item correlations depository was developed for I/O variables and construct groups. Personality measures had significantly lower inter-item correlations than other construct groups. Also, internal consistency estimates and reporting practices were evaluated over time, demonstrating an improvement in measurement precision and missing data.

Evaluation of Vestibular Function with Visual Feedback Posturography and Motorized Head Impulse Test.

Objectives: To evaluate the applicability of visual feedback posturography (VFP) for quantification of postural control, and to characterize the horizontal angular vestibulo-ocular reflex (AVOR) by use of a novel motorized head impulse test (MHIT). Methods: In VFP, subjects standing on a platform were instructed to move their center of gravity to symmetrically placed peripheral targets as fast and accurately as possible. The active postural control movements were measured in healthy subjects (n = 23), and in patients with vestibular schwannoma (VS) before surgery (n = 49), one month (n = 17), and three months (n = 36) after surgery. In MHIT we recorded head and eye position during motorized head impulses (mean velocity of 170º/s and acceleration of 1 550º/s²) in healthy subjects (n = 22), in patients with VS before surgery (n = 38) and about four months afterwards (n = 27). The gain, asymmetry and latency in MHIT were calculated. Results: The intraclass correlation coefficient for VFP parameters during repeated tests was significant (r = 0.78-0.96; p < 0.01), although two of four VFP parameters improved slightly during five test sessions in controls. At least one VFP parameter was abnormal pre- and postoperatively in almost half the patients, and these abnormal preoperative VFP results correlated significantly with abnormal postoperative results. The mean accuracy in postural control in patients was reduced pre- and postoperatively. A significant side difference with VFP was evident in 10% of patients. In the MHIT, the normal gain was close to unity, the asymmetry in gain was within 10%, and the latency was a mean ± standard deviation 3.4 ± 6.3 milliseconds. Ipsilateral gain or asymmetry in gain was preoperatively abnormal in 71% of patients, whereas it was abnormal in every patient after surgery. Preoperative gain (mean ± 95% confidence interval) was significantly lowered to 0.83 ± 0.08 on the ipsilateral side compared to 0.98 ± 0.06 on the contralateral side. The ipsilateral postoperative mean gain of 0.53 ± 0.05 was significantly different from preoperative gain. Conclusion: The VFP is a repeatable, quantitative method to assess active postural control within individual subjects. The mean postural control in patients with VS was disturbed before and after surgery, although not severely. Side difference in postural control in the VFP was rare. The horizontal AVOR results in healthy subjects and in patients with VS, measured with MHIT, were in agreement with published data achieved using other techniques with head impulse stimuli. The MHIT is a non-invasive method which allows reliable clinical assessment of the horizontal AVOR.

Teacher praise and feedback and students’ perceptions of the classroom environment

The relationships between teacher praise and feedback, and students’ perceptions of the classroom environment were investigated in six rural elementary schools (n 5 747). The Teacher Feedback Scale and My Classroom Scale were developed as part of this study and used to collect the data. Structural equation modelling was used to test a hypothesised model. The results indicated that negative teacher feedback and effort feedback were both related to students’ relationships with their teachers, while ability feedback was associated with perceptions of the classroom environment. Praise was not related to classroom environment or teacher–student relationships. Significant age and gender differences were found. Additionally, differences were found between students who were satisfied with their classroom and those who were dissatisfied. Satisfied students received more general praise, general ability feedback, effort feedback and less negative teacher feedback when compared to dissatisfied students. Research studies have emphasised the influence of signicicant adults (teachers and parents) on students’ personal development (Porlier et al., 1999) and the importance of significant others’ verbal statements when directed at children (Burnett, 1996a). The relationships between negative and positive statements made by teachers, parents, peers and siblings and children’s self-talk have been investigated (Burnett, 1996a) and positive statements (praise) have been found to be more beneficial than verbal criticism (Burnett, 1999). The quality of life in the classroom in recent times has been considered of great importance to students (Thorp et al., 1994) and this is recognised by Baker (1999) who reported a relationship between students’ satisfaction with the learning environment, and differential teacher feedback and praise. This study investigated the relationships between teacher praise and feedback, and how students perceived their classroom and their relationship with their teacher.

A measurement instrument for process modeling research : development, test and procedural model

Process modeling is an emergent area of Information Systems research that is characterized through an abundance of conceptual work with little empirical research. To fill this gap, this paper reports on the development and validation of an instrument to measure user acceptance of process modeling grammars. We advance an extended model for a multi-stage measurement instrument development procedure, which incorporates feedback from both expert and user panels. We identify two main contributions: First, we provide a validated measurement instrument for the study of user acceptance of process modeling grammars, which can be used to assist in further empirical studies that investigate phenomena associated with the business process modeling domain. Second, in doing so, we describe in detail a procedural model for developing measurement instruments that ensures high levels of reliability and validity, which may assist fellow scholars in executing their empirical research.

A virtual test assessment tool for second year signal processing students

The development and use of a virtual assessment tool for a signal processing unit is described. It allows students to take a test from anywhere using a web browser to connect to the university server that hosts the test. While student responses are of the multiple choice type, they have to work out problems to arrive at the answer to be entered. CGI programming is used to verify student identification information and record their scores as well as provide immediate feedback after the test is complete. The tool has been used at QUT for the past 3 years and student feedback is discussed. The virtual assessment tool is an efficient alternative to marking written assignment reports that can often take more hours than actual lecture hall contact from a lecturer or tutor. It is especially attractive for very large classes that are now the norm at many universities in the first two years.

Music selection using a touch screen interface : effect of auditory and visual feedback on driving and usability

This study aimed to examine the effects on driving, usability and subjective workload of performing music selection tasks using a touch screen interface. Additionally, to explore whether the provision of visual and/or auditory feedback offers any performance and usability benefits. Thirty participants performed music selection tasks with a touch screen interface while driving. The interface provided four forms of feedback: no feedback, auditory feedback, visual feedback, and a combination of auditory and visual feedback. Performance on the music selection tasks significantly increased subjective workload and degraded performance on a range of driving measures including lane keeping variation and number of lane excursions. The provision of any form of feedback on the touch screen interface did not significantly affect driving performance, usability or subjective workload, but was preferred by users over no feedback. Overall, the results suggest that touch screens may not be a suitable input device for navigating scrollable lists.

Developing sustainable IT-related capabilities : instrument development and test

Research has established that firms' IT-related capabilities at a point in time explain IT-related performance differences across firms. IT resources, however, are dynamic, and evolve at an exponential rate. This means we need to understand how to sustain firms' existing capabilities to leverage opportunities offered by new IT resources. Wet suggests a higher-level resource that can sustain firms' existing IT-related capabilities. Second, we report on the development of a valid and reliable measurement instrument for measuring this higher-level resource in four stages, which includes expert feedback and a field test. The validated instrument would be useful in extending the IT business value studies to investigate how firms can sustain their IT-related capabilities. This effort will provide a deeper understanding of how firms can secure sustainable IT-related business value from their acquired IT resources.

Design and test of a custom instrumented leg press for injury and recovery intervention

There is an increasing desire and emphasis to integrate assessment tools into the everyday training environment of athletes. These tools are intended to fine-tune athlete development, enhance performance and aid in the development of individualised programmes for athletes. The areas of workload monitoring, skill development and injury assessment are expected to benefit from such tools. This paper describes the development of an instrumented leg press and its application to testing leg dominance with a cohort of athletes. The developed instrumented leg press is a 45° reclining sled-type leg press with dual force plates, a displacement sensor and a CCD camera. A custom software client was developed using C#. The software client enabled near-real-time display of forces beneath each limb together with displacement of the quad track roller system and video feedback of the exercise. In recording mode, the collection of athlete particulars is prompted at the start of the exercise, and pre-set thresholds are used subsequently to separate the data into epochs from each exercise repetition. The leg press was evaluated in a controlled study of a cohort of physically active adults who performed a series of leg press exercises. The leg press exercises were undertaken at a set cadence with nominal applied loads of 50%, 100% and 150% of body weight without feedback. A significant asymmetry in loading of the limbs was observed in healthy adults during both the eccentric and concentric phases of the leg press exercise (P < .05). Mean forces were significantly higher beneath the non-dominant limb (4–10%) and during the concentric phase of the muscle action (5%). Given that symmetrical loading is often emphasized during strength training and remains a common goal in sports rehabilitation, these findings highlight the clinical potential for this instrumented leg press system to monitor symmetry in lower-limb loading during progressive strength training and sports rehabilitation protocols.

Frequent (flier) frustration and the dark side of word‐of‐web: exploring online dysfunctional behavior in online feedback forums

Purpose Following the perspective of frustration theory customer frustration incidents lead to frustration behavior such as protest (negative word‐of‐mouth). On the internet customers can express their emotions verbally and non‐verbally in numerous web‐based review platforms. The purpose of this study is to investigate online dysfunctional customer behavior, in particular negative “word‐of‐web” (WOW) in online feedback forums, among customers who participate in frequent‐flier programs in the airline industry. Design/methodology/approach The study employs a variation of the critical incident technique (CIT) referred to as the critical internet feedback technique (CIFT). Qualitative data of customer reviews of 13 different frequent‐flier programs posted on the internet were collected and analyzed with regard to frustration incidents, verbal and non‐verbal emotional effects and types of dysfunctional word‐of‐web customer behavior. The sample includes 141 negative customer reviews based on non‐recommendations and low program ratings. Findings Problems with loyalty programs evoke negative emotions that are expressed in a spectrum of verbal and non‐verbal negative electronic word‐of‐mouth. Online dysfunctional behavior can vary widely from low ratings and non‐recommendations to voicing switching intentions to even stronger forms such as manipulation of others and revenge intentions. Research limitations/implications Results have to be viewed carefully due to methodological challenges with regard to the measurement of emotions, in particular the accuracy of self‐report techniques and the quality of online data. Generalization of the results is limited because the study utilizes data from only one industry. Further research is needed with regard to the exact differentiation of frustration from related constructs. In addition, large‐scale quantitative studies are necessary to specify and test the relationships between frustration incidents and subsequent dysfunctional customer behavior expressed in negative word‐of‐web. Practical implications The study yields important implications for the monitoring of the perceived quality of loyalty programs. Management can obtain valuable information about program‐related and/or relationship‐related frustration incidents that lead to online dysfunctional customer behavior. A proactive response strategy should be developed to deal with severe cases, such as sabotage plans. Originality/value This study contributes to knowledge regarding the limited research of online dysfunctional customer behavior as well as frustration incidents of loyalty programs. Also, the article presents a theoretical “customer frustration‐defection” framework that describes different levels of online dysfunctional behavior in relation to the level of frustration sensation that customers have experienced. The framework extends the existing perspective of the “customer satisfaction‐loyalty” framework developed by Heskett et al.